feat(fumadb): add aggregate and keyset query support by aryasaatvik · Pull Request #1119 · UsefulSoftwareCo/executor

aryasaatvik · 2026-06-24T12:18:16Z

Summary

Add policy-scoped JSON-document aggregation and keyset pagination to FumaDB, with memory and Drizzle implementations, and expose both through indexed plugin-storage collections.

API

const byStatus = yield* runs.aggregate.groupCount({ field: "status" });
const latency = yield* runs.aggregate.stats({
  field: "durationMs",
  percentiles: [0.5, 0.95],
});
const page = yield* runs.queryKeyset({
  where: { status: "completed" },
  orderBy: [{ field: "startedAt", direction: "desc", valueType: "number" }],
  limit: 50,
});

plugin collection aggregate/queryKeyset
  -> validate declared indexed fields
  -> translate filters and apply table read policies
  -> FumaDB JSON aggregation or keyset page
  -> memory evaluation or SQLite/Postgres pushdown

Memory and SQL adapters share filter and comparison semantics.
Drizzle handles quoted paths, escaped LIKE, SQL nulls, percentiles, and nullable cursors.
Plugin-storage keys provide deterministic page tiebreakers.
Unsupported adapter capabilities fail explicitly.

Validation

Full FumaDB, SDK, and OpenAPI package suites passed.
Memory and SQLite cover aggregates, policies, filter parity, percentiles, and nullable keysets.
Affected package typechecks, format, and lint passed.
Postgres SQL generation is not yet exercised by a local Postgres harness.

Execution history delivery map

Prerequisites:

feat(execution): add execution observer foundation #1097 — Execution observer foundation
feat(fumadb): add aggregate and keyset query support #1119 — Aggregate and keyset queries 👈 current

Follow-up PR-sized diffs:

Core persistence and read API
- Runs console UI
  - Cloudflare service-token actors

Execution history is the proven consumer for this query surface, not part of this PR. View the complete fork comparison.

greptile-apps · 2026-06-24T12:28:03Z

Greptile Summary

This PR adds SQL-pushed JSON-document aggregate primitives and keyset pagination to FumaDB, exposing them through the plugin storage SDK facade. All previous review concerns (empty or filter semantics, explicit operator mapping, SQLite percentile documentation) have been addressed in this revision.

FumaDB core: New jsonCount, jsonGroupCount, jsonTimeBuckets, jsonStats, and jsonPage methods on AbstractQuery, backed by shared in-memory evaluation helpers and Drizzle SQLite/Postgres pushdown with null-aware keyset cursor predicates and dialect-specific JSON path quoting/LIKE escaping.
Plugin storage facade: New aggregate.{count,groupCount,timeBuckets,stats} and queryKeyset(...) methods on collection facades, translating indexed-field where conditions into JsonFilter predicates while table-read policies enforce owner/tenant scoping before every adapter call.
Test coverage: Memory and SQLite harnesses exercising count, group count, time buckets, stats, keyset pagination with null sort values, LIKE-wildcard parity, path-quoting edge cases, and policy enforcement for the new operations.

Confidence Score: 5/5

Safe to merge — all five new query operations apply table read policies before reaching the adapter, and the empty-or / explicit-operator issues from the previous review are resolved.

The implementation is correct and well-tested across memory, SQLite, and policy harnesses. The null-aware keyset cursor logic is exercised end-to-end (including the nullable-sort-column truncation regression). JSON path quoting, LIKE escaping, and empty composite filter semantics all have explicit parity tests between the memory and Drizzle adapters.

No files require special attention. The most complex file (drizzle/query.ts) is fully covered by the aggregate.test.ts harness for SQLite, and the Postgres-specific paths (percentile_cont, nulls last/first) are intentionally deferred to a future Postgres test harness as noted in the PR description.

Important Files Changed

Filename	Overview
packages/core/fumadb/src/adapters/drizzle/query.ts	Adds jsonCount/jsonGroupCount/jsonTimeBuckets/jsonStats/jsonPage adapter hooks with dialect-aware JSON path quoting, LIKE wildcard escaping with correct ESCAPE clause, null-aware keyset cursor predicates, and Postgres percentile_cont pushdown. Empty `or` filter correctly returns `1 = 0`.
packages/core/fumadb/src/query/orm/index.ts	Adds `compileScopedWhere` helper that applies read policies before every new JSON operation, `requireJsonOp` that throws clearly for unsupported adapters. The `scopedWhere === false` short-circuit correctly returns zero/empty results when a policy statically denies access.
packages/core/sdk/src/executor.ts	Adds CoreDb plumbing for the five JSON operations, `pluginStorageWhereToJsonFilter` translation, `pluginStorageInvalidLimitError` guard, and the `queryKeyset`/`aggregate` facade implementations. `nextCursor` is correctly set only when `entries.length >= limit`. `pluginStorageJsonCompareOperators` is now an explicit mapping.
packages/core/fumadb/src/query/aggregate-eval.ts	New file with pure in-memory evaluation helpers: extractJsonPath, coerceJsonValue, matchesJsonFilter, bucketFloor, computePercentiles (Postgres percentile_cont linear interpolation), and compareNullableAscending.
packages/core/fumadb/src/adapters/memory/index.ts	Adds all five JSON operations using the shared aggregate-eval helpers. The jsonPage implementation correctly applies null-aware compareNullableAscending sort and cursor filter, consistent with the Drizzle adapter's SQL null ordering semantics.
packages/core/fumadb/src/query/aggregate.ts	New public type definitions for all JSON aggregate and keyset pagination primitives, split into adapter-facing options and public options. Clean separation of concerns.
packages/core/fumadb/src/query/aggregate.test.ts	New test file running the full aggregate + keyset suite against memory and SQLite harnesses, covering null SQL three-valued logic, empty composite filters, nullable sort column keyset pagination, LIKE wildcard parity, and path-quoting edge cases.
packages/core/sdk/src/plugin-storage-aggregate.test.ts	New integration test running the full plugin storage aggregate + keyset suite against the SQLite pushdown path, including multi-page keyset cursor pagination with a JSON filter.
packages/core/fumadb/src/query/table-policy.test.ts	Extended with a test confirming read policies are applied before all JSON aggregate and keyset operations, and that a denied context correctly returns zero count and empty page results.
packages/core/sdk/src/plugin-storage.ts	Adds public facade types: PluginStorageAggregateFilter, PluginStorageQueryKeysetInput, PluginStorageKeysetPage, and PluginStorageAggregateFacade. queryKeyset and aggregate added to PluginStorageCollectionFacade.
packages/core/sdk/src/fuma-runtime.ts	Adds the five new JSON operation forwards to makeSafeFumaQuery, consistent with the existing operation forwarding pattern.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Plugin as Plugin Code
    participant Facade as PluginStorageFacade
    participant CoreDb as CoreDb
    participant ORM as toORM (orm/index.ts)
    participant Adapter as Drizzle/Memory Adapter

    Plugin->>Facade: "storage.runs.aggregate.groupCount({ field, where })"
    Facade->>Facade: validate indexed fields
    Facade->>Facade: pluginStorageWhereToJsonFilter(where)
    Facade->>CoreDb: "jsonGroupCount(plugin_storage, { column, where, filter })"
    CoreDb->>ORM: jsonGroupCount(name, options)
    ORM->>ORM: requireJsonOp + compileScopedWhere
    ORM->>ORM: applyReadPolicies
    ORM->>Adapter: "jsonGroupCount(table, { column, where, filter, path })"
    Adapter-->>ORM: JsonGroupCountRow[]
    ORM-->>Facade: JsonGroupCountRow[]
    Facade-->>Plugin: PluginStorageGroupCount[]

    Plugin->>Facade: "storage.runs.queryKeyset({ orderBy, cursor, limit })"
    Facade->>Facade: validate indexed fields + limit
    Facade->>CoreDb: "jsonPage(plugin_storage, { orderBy, keyColumn, cursor, limit })"
    CoreDb->>ORM: jsonPage(name, options)
    ORM->>ORM: requireJsonOp + compileScopedWhere
    ORM->>ORM: applyReadPolicies
    ORM->>Adapter: "jsonPage(table, { orderBy, keyColumn, keyDirection, cursor, limit })"
    Note over Adapter: null-aware cursor predicate OR-terms
    Adapter-->>ORM: Row[]
    ORM-->>Facade: Row[]
    Facade->>Facade: build nextCursor from last entry
    Facade-->>Plugin: "{ entries, nextCursor }"

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Plugin as Plugin Code
    participant Facade as PluginStorageFacade
    participant CoreDb as CoreDb
    participant ORM as toORM (orm/index.ts)
    participant Adapter as Drizzle/Memory Adapter

    Plugin->>Facade: "storage.runs.aggregate.groupCount({ field, where })"
    Facade->>Facade: validate indexed fields
    Facade->>Facade: pluginStorageWhereToJsonFilter(where)
    Facade->>CoreDb: "jsonGroupCount(plugin_storage, { column, where, filter })"
    CoreDb->>ORM: jsonGroupCount(name, options)
    ORM->>ORM: requireJsonOp + compileScopedWhere
    ORM->>ORM: applyReadPolicies
    ORM->>Adapter: "jsonGroupCount(table, { column, where, filter, path })"
    Adapter-->>ORM: JsonGroupCountRow[]
    ORM-->>Facade: JsonGroupCountRow[]
    Facade-->>Plugin: PluginStorageGroupCount[]

    Plugin->>Facade: "storage.runs.queryKeyset({ orderBy, cursor, limit })"
    Facade->>Facade: validate indexed fields + limit
    Facade->>CoreDb: "jsonPage(plugin_storage, { orderBy, keyColumn, cursor, limit })"
    CoreDb->>ORM: jsonPage(name, options)
    ORM->>ORM: requireJsonOp + compileScopedWhere
    ORM->>ORM: applyReadPolicies
    ORM->>Adapter: "jsonPage(table, { orderBy, keyColumn, keyDirection, cursor, limit })"
    Note over Adapter: null-aware cursor predicate OR-terms
    Adapter-->>ORM: Row[]
    ORM-->>Facade: Row[]
    Facade->>Facade: build nextCursor from last entry
    Facade-->>Plugin: "{ entries, nextCursor }"

_{Reviews (3): Last reviewed commit: "fix(fumadb): preserve empty or filter se..." | Re-trigger Greptile}

## Summary Mirrors the review-hardening deltas from upstream [UsefulSoftwareCo#1119](UsefulSoftwareCo#1119) onto `dev`. `dev` already contains the broader FumaDB aggregate and keyset query feature, so this PR only carries the remaining drift from the upstream review loop. ## Changes - Preserve memory and Drizzle parity for empty composite filters by compiling empty JSON `or` filters to a constant false SQL predicate. - Add regression coverage for empty `or` and empty `and` filters across the aggregate harness. - Replace nested plugin-storage operator selection with an explicit typed JSON compare-operator map. - Document SQLite percentile behavior on the public FumaDB and plugin-storage stats inputs. ## Intentional Differences From Upstream UsefulSoftwareCo#1119 - This PR does not re-add the full aggregate and keyset implementation because `dev` already has that feature surface. - This PR does not touch the OpenAPI storage facade mock because `dev` already has the mock shape needed by the expanded collection facade. - This PR has no changeset because it only mirrors fixes to an existing `dev` feature surface. ## Tests - `bun run bootstrap` - `bun run --cwd packages/core/fumadb test -- src/query/aggregate.test.ts src/query/table-policy.test.ts` - `bun run --cwd packages/core/sdk test -- src/plugin-storage-aggregate.test.ts src/plugin-storage.test.ts` - `bun run --cwd packages/core/fumadb typecheck` - `bun run --cwd packages/core/sdk typecheck` - `bun run typecheck` - `./node_modules/.bin/oxfmt --check packages/core/fumadb/src/adapters/drizzle/query.ts packages/core/fumadb/src/query/aggregate.test.ts packages/core/fumadb/src/query/aggregate.ts packages/core/sdk/src/executor.ts packages/core/sdk/src/plugin-storage.ts`

aryasaatvik · 2026-08-27T17:09:23Z

Rebased onto current main and refreshed at c9e1fb3. Branch-focused FumaDB, SDK, and OpenAPI tests; full package suites; format; lint; typecheck; unit tests; and 35 of 36 current GitHub checks pass. The sole red check is unrelated selfhost E2E infrastructure: admin-users-pager.test.ts uses a global getByRole button query for Next, which matched both the pager button and a generated user-row button named Next, causing a Playwright strict-mode failure. The affected storage tests and the previously failing OAuth scenario pass locally.

Add reusable JSON-document aggregate and keyset pagination primitives across FumaDB memory and Drizzle adapters. Expose the pushdown through plugin storage aggregate and queryKeyset facades with focused coverage for SQLite parity, null handling, path escaping, policy scoping, and unsupported adapters.

greptile-apps Bot reviewed Jun 24, 2026

View reviewed changes

Comment thread packages/core/sdk/src/executor.ts Outdated

Comment thread packages/core/fumadb/src/adapters/drizzle/query.ts

Comment thread packages/core/fumadb/src/adapters/drizzle/query.ts

Comment thread packages/core/fumadb/src/query/aggregate.test.ts

greptile-apps Bot reviewed Jun 24, 2026

View reviewed changes

Comment thread packages/core/fumadb/src/adapters/drizzle/query.ts

aryasaatvik mentioned this pull request Jun 24, 2026

fix(fumadb): mirror aggregate review hardening aryasaatvik/executor#62

Merged

aryasaatvik force-pushed the contrib/fumadb-aggregate-keyset branch 5 times, most recently from 3852cc4 to c9e1fb3 Compare August 27, 2026 17:04

This was referenced Aug 27, 2026

feat(execution-history): add plugin-owned run persistence aryasaatvik/executor#81

Closed

feat(execution): add execution observer foundation #1097

Open

aryasaatvik force-pushed the contrib/fumadb-aggregate-keyset branch 2 times, most recently from da98e37 to 6d210fe Compare August 29, 2026 07:17

aryasaatvik added 4 commits August 29, 2026 13:00

fix(fumadb): address aggregate review notes (greptile)

5fa2d6e

fix(openapi): update storage facade mock (greptile)

9d9bda1

fix(fumadb): preserve empty or filter semantics (greptile)

58c8a98

aryasaatvik force-pushed the contrib/fumadb-aggregate-keyset branch from 6d210fe to 58c8a98 Compare August 29, 2026 07:32

Navigation Menu

Uh oh!

feat(fumadb): add aggregate and keyset query support - #1119

feat(fumadb): add aggregate and keyset query support#1119
aryasaatvik wants to merge 4 commits into
UsefulSoftwareCo:mainfrom
aryasaatvik:contrib/fumadb-aggregate-keyset

aryasaatvik commented Jun 24, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jun 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aryasaatvik commented Aug 27, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

aryasaatvik commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

API

Validation

Execution history delivery map

Uh oh!

greptile-apps Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aryasaatvik commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

aryasaatvik commented Jun 24, 2026 •

edited

Loading

greptile-apps Bot commented Jun 24, 2026 •

edited

Loading

aryasaatvik commented Aug 27, 2026 •

edited

Loading